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Abstract: This study assesses the influence of a two-year language 
program evaluation on program directors and faculty career 
development. The study makes use of mixed-paradigms (positivism 
and qualitative interpretive), mixed-strategies (survey research and 
qualitative evaluation), one-way analysis of variance (ANOVA) and a 
post-hoc test of multiple comparisons (Scheffe). The findings indicate 
that imposed program evaluation experiences help faculty members 
advance their career skills in terms of course planning, classroom 
teaching, learning assessment, classroom research, and coping with 
career pressure. The findings also indicate significant improvement in 
program director academic and administrative career skills. 
Moreover, the findings do not show inter-program differences 
regarding the program evaluation impact on faculty and program 
director career skills. The study recommends program evaluation as 
an effective and systematic approach to program stakeholder career 
development. 


Introduction 

Although career development has been essential for institutional and program 
development (Lee, Maerten-Rivera, Penfield, LeRoy & Secada, 2008), not many program 
directors, faculty members and other staff advance their professional skills beyond initial 
training. Not only does this result in stakeholder professional underdevelopment but also 
causes poor program learning outcomes. Concerns have therefore been expressed about 
stakeholders’ career development opportunities in the workplace (Shawer, 2010a; Shawer, 
Gilmore & Banks-Joseph, 2008). 

Program evaluation has recently been a promising approach for addressing the 
concerns raised about career development opportunities of program stakeholders in the 
workplace (Byrnes, 2008; Chase, 2006; Shawer, 2011). Empirical research seems also to take 
the same direction about the positive influence of program evaluation experiences on 
stakeholder career development (e.g., Carsten-Wickham, 2008; Gorsuch, 2009). Program 
evaluation emerges as a program and career development strategy for providing not only 
information about program improvement, change or even termination, but also creates a 
context for stakeholder career development in the workplace. Moreover, program evaluation 
has been the central tool in assessing program weaknesses and strength so that programs can 
deliver what they promise and justify their existence (Clarke, 1999; Norris, 2006; Stake, 
2011). The present study, therefore, sought to address career development concerns through 
examining the impact of program evaluation on faculty and program directors’ career 
development opportunities in three language-education programs at King Saud University. 
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The first (Language and Culture) is an intensive program designed to develop student 
language proficiency so that they can pursue college education. It provides courses that cover 
the four language skills (reading, listening, writing and speaking), vocabulary development 
and phonetics alongside grammar and computer applications in language learning. The 
second program (Teacher Training) targets inservice teachers who seek to continue their 
professional development. It provides courses on second and foreign language learning 
theories, applied linguistics, sociolinguistics, syllabus design, teaching methods, language 
testing, and use of technology in second and foreign language teaching. The third program 
(Teacher Preparation) targets prospective teachers who aspire to be second or foreign 
language teachers. The courses include introduction to linguistics, phonology, syntax, 
teaching methods, computer-assisted language learning, second language acquisition, a 
practicum, applied linguistics, contrastive and error analysis, language testing, materials 
design and development, and research in applied linguistics. 


The Evaluation Program 

An evaluation program has been designed to collect and analyse data against the 
National Commission for Academic Accreditation and Assessment (NCAAA) 11 standards in 
each of the three programs. The NCAAA standards require collection of evidence about each 
program: (1) mission, goals and objectives, (2) governance and administration, (3) 
management of quality assurance and improvement, (4) teaching and learning, (5) student 
administration and support services, (6) learning resources, (7) facilities and equipment, (8) 
financial planning and management, (9) employment, (10) research, and (11) relationships 
with the community. These standards address all program components. The three programs 
also collected their data by the same scales standardized by the NCAAA. 

The evaluation design followed Salabarria-Pena, Apt and Walsh’s (2007) six steps. 
First, we briefed implementer stakeholders (faculty members) and decision-maker 
stakeholders (program directors) of the evaluation to encourage their participation, support 
evaluation efforts and advocate evaluation findings. All faculty members and program 
directors were part of the evaluation team to mark changes in their skills due to their 
participation. A second step involved a program description to understand program focus and 
priorities, develop goals and objectives, get familiar with program elements, and link inputs, 
activities and outcomes in a program logic model. A logic model provides a picture or 
diagrammatic representation about how programs work (Kirkpatrick, 2001; Salabarria-Pena 
et ah, 2007). It is “a logical series of statements that link the problems your program is 
attempting to address (conditions), how it will address them (activities), and what are the 
expected results (outcomes)” (Bliss & Emshoff, 2002, p. 6). All faculty members across the 
three programs participated in this as well as subsequent stages. 

In the third step, we focused evaluation through detennining resources and personnel, 
writing evaluation questions and deciding which activities to evaluate (King, Morris & Fitz- 
Gibbon, 1987; Patton, 1997; Stecher, 1987). The fourth step concerned collecting credible 
evidence by specifying indicators (also performance measures) to answer evaluation 
questions and specify data gathering sources and methods. We defined each indicator as “a 
specific, observable, and measurable accomplishment or change that shows whether progress 
has been made toward achieving a specific program output or outcome” in our program logic 
model (Salabarria-Pena et al., 2007, p. 238). Program outputs refer to the number of 
clients/products who completed a program, whereas program outcomes refer to the benefits 
of or changes in those clients/products (McNamara, 2012; Weiss, 1972). 
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We made each indicator specific by providing a clear description of the target 
behaviour we sought to measure. For example, instead of just saying the program has 
qualified faculty, we stated ‘faculty members have PhD qualifications in English foreign 
language teaching from an accredited university’. We also made each indicator observable 
through ensuring each one has actions or changes. An example of an indicator that meets this 
criterion was, ‘each program keeps a written document of completion rates’. Moreover, each 
indicator was made measurable by quantifying target changes in numerical terms, as in ‘the 
teacher to student ratio should not exceed 1 to 20’. We developed indicators through 
consultations with stakeholders, reviewing evaluation questions and using program logic 
model as a template to develop and relate indicators to stated outcomes. 

We linked indicators to activities, outputs and outcomes in our logic model to monitor 
progress through use of outcome and process indicators. We used outcome indicators to 
“measure whether progress was made toward achieving ... outcomes,” whereas process 
indicators were used to “measure whether progress is made toward achieving 
implementation” (Salabarria-Pena et ah, 2007, p. 232). ‘Faculty members teach weekly topics 
according to plan in the course specification and justify untaught topics’ is an example of a 
process indicator we used, whereas ‘faculty members’ ability to write doable weekly plans 
improves’ is one example of outcome indicators. We then determined potential data sources 
for each indicator and linked data sources and collection methods to indicators and evaluation 
questions (Salabarria-Pena et ah, 2007). 

In the fifth step we analysed and synthesized evaluation findings in ways that allow 
better understanding of each program activities and components (Salabarria-Pena et ah, 2007; 
Scriven, 2011). The final step concerned use of evaluation findings and sharing lessons 
learned from evaluation with stakeholders to show how they can use evaluation results to 
modify, strengthen, and improve the program (Salabarria-Pena et ah, 2007). Through 
involving faculty members and program directors, we hoped they would improve their skills. 
For example, by asking instructors to read documents and attend discussions about how to 
develop course specifications, we expected improvement in their course planning skills. We 
expected them to be able to write precise course aims and objectives, decide on cognitive, 
affective, psychomotor and interpersonal skills in their courses. We also expected them to 
become able to align course content with teaching and assessment targets and strategies 
alongside aligning course and program learning outcomes. 


Career Development and Program Evaluation 

Career or professional development is where individuals continue to advance their 
knowledge and skills during their careers (Shawer, 2010a). This involves “all types of 
professional learning undertaken by teachers beyond the point of initial training” (Craft, 

1996, p. 6). Career development, however, has been mistakenly confined to institution- 
initiated fonnal “interventions and training to direct the evolution in professional behaviour 
in a more desirable way” (Kelchtermans & Vandenberghe, 1994, p. 45). Fortunately, career 
development has recently become a process of career-long learning in the workplace 
(Anderson & Olsen, 2006). As such, career development involves those “ongoing fonnal and 
informal learning activities through which professionals continue to advance their 
professional competence so that they can improve their practices and profession” (Shawer, 
2010a, p. 598). Since professionals better advance their career skills through learning from 
actual experiences in the workplace (Schon, 1983), program evaluation has recently 
materialized a rich context where faculty and staff advance their professional development in 
action (Byrnes, 2008; Chase, 2006). 
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Program evaluation is “an infonnation-gathering and -interpreting endeavor that 
attempts to answer a specified set of questions about a program’s perfonnance and 
effectiveness” (Rossi, Freeman & Lipsey, 1999, p. 62). Program evaluation has therefore 
become an indispensable mechanism for improving language-education as well as generic- 
education programs (Shawer, 2010a, 2012). It enables programs to determine and achieve 
their objectives and address new developments through assessing program strengths and 
weaknesses (Bernhardt, 2006; Patton, 1990; Sullivan, 2006). Moreover, program evaluation 
demonstrates how and why programs respond to stakeholders. Being so, program evaluation 
provides evidence about whether a program should continue or simply shut down. Because 
“programs exist in order to change, enrich, enhance, extend, or improve the lives of 
participants and, by extension, the quality of life in society as a whole,” program evaluation 
allows the public and higher-education institutions to ascertain that programs deliver what 
they promise (Norris, 2006, p. 577). 

Program evaluation also answers accrediting bodies why programs should continue. 
Despite such importance, program evaluation has been concerned solely with doing rather 
than using program evaluation (Norris, 2006). Unfortunately, many programs direct all effort 
and resources to conduct program evaluation while think little of assessing the value of the 
evaluation process to program stakeholders (Norris, 2009). How programs and stakeholders 
benefit from the evaluation process remains somewhat absent in research (Elder, 2009; Kiely 
& Rea-Dickins, 2005). 


Language-Education Programs 

Program evaluation generates important infonnation about language proficiency gains 
and ultimately the effectiveness of language-education programs (Ross, 2003). A language- 
education program “generally consists of a slate of courses designed to prepare students for 
some language-related endeavor” (Lynch, 1996, p. 2). Program evaluation is therefore 
essential not only to improve program performance (planning, design, implementation and 
outcomes) but also to meet institutional requirements. Through program evaluation, 
language-education programs are able to set precise program objectives, instructional 
strategies, assessment targets and allocate resources (Lynch, 1996). 

Although program evaluation helps programs demonstrate whether they address 
quality, public accountability and accreditation concerns, many program stakeholders 
consider imposed evaluation as a threat rather than an opportunity for help and improvement 
(Norris, 2006). As a result, stakeholders undertake program evaluation as an end rather than a 
means of development (Byrnes, 2006). Program evaluation should, however, be a context for 
faculty members to learn while they practice the teaching profession (Shawer, 2010b). 

Effective program evaluation in general and language program evaluation in 
particular can be a powerful improvement strategy when stakeholders use more than do 
evaluation. When implementer stakeholders think of what they can learn from each 
evaluation task rather than just carrying out evaluation activities, they are making use of 
evaluation. For example, while implementer stakeholders assess the formulation of‘statement 
of program mission,’ they leam how to write it, what use could be made of it and how it 
could be linked to an institution’s mission. When stakeholders view imposed evaluation as a 
career development strategy, it turns into an opportunity for institutional, program and 
professional development (Norris, 2006). From the very beginning, stakeholders will use 
program evaluation to improve program targets, content, teaching and learning, and 
assessment means and outcomes. Student gains in language arts (reading, writing, speaking 
and listening) in particular will dramatically increase (Lynch, 1996). 
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Evaluation of the influence of program evaluation on faculty career development has been 
somewhat absent in previous research (Norris, 2009). Fortunately, the attention has recently 
shifted to examining the value of program evaluation to program stakeholders. Although the 
relationship between program evaluation and faculty professional development was not the 
main focus by prior research, some previous studies found a positive correlation between 
language program evaluation, whether internally-motivated or externally-imposed, and 
faculty professional development (e.g., Byrnes, 2008; Carsten-Wickham, 2008; Chase, 2006; 
Gorsuch, 2009). In light of the above review, the present study sought to answer the 
following research questions: 

1. How does the program evaluation process impact on career development? 

2. How does the program evaluation process impact on career development in the three 
programs? 

Methods and Participants 

As shown in Figure 1, survey research was used to address the first research question 
by describing and interpreting the opinions of program stakeholders about the value of 
program evaluation to their career/ professional development (Cohen, Manion & Morrison, 
2011). Career development was measured in terms of course planning, classroom teaching, 
learning assessment, classroom research, and coping with career pressure. The researchers 
tested this null hypothesis to answer the first research question: program evaluation did not 
influence career development. 



Figure 1: Research Design. 

Although survey research could answer the first research question, the standardized 
responses it generated did not justify why faculty members and program directors provided 
positive or negative assessments regarding the influence of program evaluation on their 
career development. A qualitative paradigm was therefore followed to allow interactions with 
the respondents and understand their context (Figure 1). This involved using qualitative 
evaluation to assess why program stakeholders view a positive or negative influence of 
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program evaluation on their career development. Qualitative evaluation involved the 
collection, analysis and interpretation of spoken and written discourse about program 
evaluation impact in order to use the resulting information for future stakeholder career 
development (Shawer, 2012). Both quantitative and qualitative evaluations assess program 
effectiveness, including planning, implementation, instructional methods, curriculum 
materials, facilities, equipment, educators and students better than other research strategies 
(Clarke, 1999; Gall, Gall & Borg, 2006; Patton, 1990). 

Survey research was also used to answer the second research question by collecting 
stakeholder opinions in each program and comparing them to one another. The researchers 
used a cross-sectional design in particular to concurrently collect data from different faculty 
members and program directors at the three programs (Cohen et ah, 2011; Robson, 1993). 

The researchers tested this null hypothesis to address the second research question: there 
were no inter-program differences at 0.05 in career development as a result of program 
evaluation. This alternative hypothesis was, however, posed in case the null hypothesis was 
rejected: there were inter-program differences at 0.05 in career development as a result of 
program evaluation. 

As also shown in Figure 1, the researchers drew a random sample of 39 faculty 
members at the three language programs: the Language and Culture Program (group 1), the 
Teacher Training Program (group 2) and the Teacher Preparation Program (group 3). Group 
one consisted of 17 faculty members, group two involved 12 faculty members, whereas group 
three comprised 10 faculty members. Some faculty members were EFL (English as a foreign 
language) teachers, whereas some of them were ASL (Arabic as a second language) teachers. 
Both anonymity and confidentiality were assured and maintained to encourage the 
respondents to provide valid responses and to enlist their cooperation (Lester & Lester, 

2010 ). 

A rating-scales questionnaire of six sections was designed to gather the research data 
(see the appendix). Seven professors from the three programs examined the questionnaire 
content and agreed the questions met the research purpose (Bloom, Fischer & Onne, 2009; 
Shawer, 2012). The researchers then checked the questionnaire reliability for internal 
consistency to ensure the respondents had a consistent performance on all the scale’s items. 
Alpha coefficient was calculated because items carried different weights (Gall et al., 2006). 

Using SPSS (version 18), the calculation of the coefficient of internal consistency on 
23 respondents resulted in a 0.82 Cronbach's Alpha (Coakes & Steed, 2007). Gall et al. 

(2006) note that scales of coefficients above 0.80 are considered reliable. The data was first 
analyzed through descriptive statistics, including averages, percentages and standard 
deviations. One-way analysis of variance (ANOVA) was then calculated to examine group 
mean differences. A post hoc test of multiple comparisons (Scheffe) was further used to 
examine mean differences between each two groups. The quantitative results section shows 
how the researchers analyzed the questionnaire data. 

The researchers used semi-structured interviews to allow each program director and 
faculty member explain the influence of the program evaluation process on career skills. 
Three faculty members in each program expressed their views in one-to-one interviews 
whose time took between 30 to 60 minutes. Three language professors and two educational 
researchers checked the interview content. Despite asking for revising the wording and 
position of some questions, they finally agreed the questions addressed the research purpose 
(Kvale, 1996; Patton, 1990). Interview questions were subject to piloting and re-piloting until 
we made sure they were reliable. Audio recordings and accuracy of transcriptions further 
validated the responses. Interviews analysis involved coding concepts, grouping similar 
concepts under categories and fonning a narrative (Strauss & Corbin, 1998). 
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Quantitative Results 

Program Evaluation Influence on Career Development 

This section addressed the first research question (How does the program evaluation 
process impact on career development?). Before presenting the data, we explain the process 
of data analysis in Table 1 which also applies to Table 2. Table 1 shows group one responses 
to the six variables were analysed through sums, the minimum and maximum possible scores, 
percentages and averages. The sum of responses of the first variable (overall career skills 
development) was 236, the minimum score was 51 (3 (items) x 1 (minimum possible 
responses) = 3x17 (number of respondents)) while the maximum score was 255 (3 (items) x 
5 (maximum possible responses) = 15 x 17 (number of respondents)). The percent was 93 
(236 (sum of responses) 255 (maximum possible responses) x 100). 

Table 1 also shows the sum of responses of the second variable (course planning) 
was 541, the minimum score was 119 (7 (items) x 1 (minimum possible responses) = 7 x 17 
(number of respondents)) while the maximum score was 595 (7 (items) x 5 (maximum 
possible responses) = 35 x 17 (number of respondents)). The percent was 91 (541 (sum of 
responses) 595 (maximum possible responses) x 100). Moreover, the sum of responses of 
the third variable (classroom teaching) was 389, the minimum score was 85 (5 (items) x 1 
(minimum possible responses) = 5 x 17 (number of respondents)), while the maximum 
score was 425 (5 (items) x 5 (maximum possible responses) = 25 x 17 (number of 
respondents)). The percent was 92 (389 (sum of responses) 425 (maximum possible 
responses) x 100). 



Variable 

Sum 

Min. Score 

Maxim. Score 

Percentage 

Mean 

1 

OvCarSk 

236 

51 

255 

93 

13.88 

2 

CorsPlan 

541 

119 

595 

91 

31.82 

3 

ClassTea 

389 

85 

425 

92 

22.88 

4 

LearnAss 

619 

136 

680 

91 

36.41 

5 

ClassRes 

315 

68 

340 

93 

18.52 

6 

CareePres 

228 

51 

255 

89 

13.41 

7 

OVERALL 

2328 

510 

2550 

91 

136.94 


Table 1: Group 1 descriptive statistics (the language and culture program) 


As regards the fourth variable (learning assessment skills), Table 1 further shows the 
sum of responses was 619, the minimum score was 136 (8 (items) x 1 (minimum possible 
responses) = 8 x 17 (number of respondents)), while the maximum score was 680 (8 (items) 
x 5 (maximum possible responses) = 40 x 17 (number of respondents)). The percent was 91 
(619 (sum of responses) 680 (maximum possible responses) x 100). The sum of responses 
of the fifth variable (classroom research skills) was 315, the minimum score was 68 (4 
(items) x 1 (minimum possible responses) = 4 X 17 (number of respondents)), while the 
maximum score was 340 (4 (items) x 5 (maximum possible responses) = 20 x 17 (number 
of respondents). The percent was 93 (315 (sum of responses) = 340 (maximum possible 
responses) x 100). 

Concerning the sixth variable (career pressure), Table 1 shows the sum of responses 
was 228, the minimum score was 51 (3 (items) x 1 (minimum possible responses) = 3 x 17 
(number of respondents)), while the maximum score was 255 (3 (items) x 5 (maximum 
possible responses) =15x17 (number of respondents)). The percent was 89 (173 (sum of 
responses) = 200 (maximum possible responses) x 100). Finally, the sum of responses of the 
overall variable (the six variables) was 2328, the minimum score was 510 (30 (items) x 1 
(minimum possible responses) = 30 x 17 (number of respondents)) while the maximum 
score was 2550 (30 (items) x 5 (maximum possible responses) = 150 x 17 (number of 
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respondents)). The percent was 91 (2328 (sum of responses) 2550 (maximum possible 
responses) x 100). All averages were calculated using SPSS (version 18). 

As regards the actual findings, Table 1 indicates that group one responses to the six 
questionnaire sections were high. Faculty responses (93%) show the program evaluation 
process improved faculty overall career skills (first variable). Their responses mean the 
evaluation process improved their overall teaching skills, domain knowledge and their skills 
of assessing classroom learning outcomes. 

Likewise, faculty responses (91%) show the evaluation process improved their course 
planning skills (second variable). Thanks to the evaluation process, the teaching faculty 
developed their skills in regards to setting precise course objectives, preparing the course 
specification and report as well as aligning course and program learning outcomes. Faculty 
members also developed their course planning skills in terms of fitting course topics to 
semester teaching hours, designing course materials and evaluating and selecting course 
materials. 


© 

Z 

Variable 

The teacher training program 

The teacher preparation program 

Sum 

Min. 

Score 

Maxim. 

Score 

S 

Mean 

Sum 

Min. 

Score 

Maxim. 

Score 

5? 

Mean 

i 

OvCarSk 

166 

36 

180 

92 

13.83 

140 

30 

150 

93 

14.00 

2 

CorsPlan 

284 

84 

420 

86 

32.00 

325 

70 

350 

93 

32.50 

3 

ClassTea 

276 

60 

300 

92 

23.00 

231 

50 

250 

92 

23.10 

4 

LearnAss 

438 

96 

480 

91 

36.50 

371 

80 

400 

93 

37.10 

5 

ClassRes 

220 

48 

240 

92 

18.33 

188 

40 

200 

94 

18.80 

6 

CareePres 

165 

36 

180 

92 

13.75 

139 

30 

150 

93 

13.90 

7 

OVERALL 

1649 

360 

1800 

92 

137.41 

1394 

300 

1500 

93 

139.40 


Table 2: Group 2 and 3 descriptive statistics 


Table 1 further shows 92 % of the responses indicate the evaluation process improved 
faculty classroom teaching skills (third variable). The teaching faculty became more able to 
spell out their teaching philosophy, set precise lesson objectives, align course and program 
learning objectives, align classroom teaching with exam content and use various teaching 
strategies. So was the case with learning assessment skills (fourth variable) since 91% of the 
responses confirm the positive influence of program evaluation on faculty skills of writing 
test items, test design and checking test reliability and validity. Their learning assessment 
skills also improved regarding analyzing and interpreting test results, creating item banks and 
using various assessment tools. 

Moreover, Table 1 indicates improvement in the teaching faculty classroom research 
skills (fifth variable). Most responses (93%) show improvement in their ability to survey 
student opinions and analyze and interpret survey data alongside incorporating student 
assessments into course reports. In the same vein, the evaluation process improved the 
teaching faculty ability to cope with their career pressure (sixth variable). The majority of 
responses (89%) point to improvement in their ability to work under pressure, cope with team 
work, and address student concerns. Finally, the combined faculty responses of all variables 
(91%) indicate improvement in the faculty overall career skills as a result of the program 
evaluation process. As summarized in Table 2, the responses of the other two groups were in 
consonance with those of group one. 

The three group findings indicate that the imposed program review process resulted in 
significant improvement in the faculty career skills. The null hypothesis stating no influence 
of program evaluation on developing faculty career skills was therefore rejected while 
accepting the alternative hypothesis that indicates a positive influence of program evaluation 
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on career skills. Having found close findings across the three programs, the researchers went 
further to test group means for significance in the following section. 


Variable 

Group 

N 

Mean 

S.D 

Variable 

Group 

N 

Mean 

S.D 


1 

17 

13.88 

1.53 

LearnAss 

1 

17 

36.41 

3.74 

OvCarSk 

2 

12 

13.83 

1.52 


2 

12 

36.50 

3.70 


3 

10 

14.00 

1.33 


3 

10 

37.10 

3.14 


Total 

39 

13.89 

1.44 


Total 

39 

36.61 

3.50 


1 

17 

31.82 

3.14 

ClassRes 

1 

17 

18.52 

1.80 

CorsPlan 

2 

12 

32.00 

3.19 


2 

12 

18.33 

1.37 


3 

10 

32.50 

2.83 


3 

10 

18.80 

1.13 


Total 

39 

32.05 

3.01 


Total 

39 

18.53 

1.50 


1 

17 

22.88 

1.86 

CareePres 

1 

17 

13.41 

1.41 

ClassTea 

2 

12 

23.00 

1.90 


2 

12 

13.75 

1.138 


3 

10 

23.10 

1.72 


3 

10 

13.90 

.31 


Total 

39 

22.97 

1.79 


Total 

39 

13.64 

1.13 


1 

17 

136.94 

12.66 






OVERALL 

2 

12 

137.41 

11.24 







3 

10 

139.40 

8.42 







Total 

39 

137.71 

11.04 







Table 3: Group descriptive statistics 


Inter-program Differences in Career Development 

This section answered the second research question (How does the program 
evaluation process impact on career development in the three programs?). Table 3 shows very 
similar inter-program averages and standards deviations. These descriptive statistics indicate 
the program evaluation process not only improved the teaching faculty career skills but also 
indicate no differences between the three programs. We therefore moved to test group means 
for significance to make sure such a typical impact of program evaluation on faculty career 
development is real through one-way ANOVA. 


Variables 

Levene Statistic 

dfl 

df2 

Sig. 

OvCarSk 

.258 

2 

36 

.774 

CorsPlan 

.056 

2 

36 

.946 

ClassTea 

.149 

2 

36 

.862 

LearnAss 

.289 

2 

36 

.750 

ClassRes 

.774 

2 

36 

.469 

CareePres 

6.397 

2 

36 

.004 

OVERALL 

.659 

2 

36 

.523 


Table 4: Levene's test of equality of error variances 


Before running ANOVA tests, the data was screened to ensure it meets the ANOVA 
assumptions of statistical analysis. The homogeneity assumption was first checked by the 
Box’s M test that was not significant (p. > .05). Table 4 shows the ANOVA assumption of 
homogeneity was further examined by the Levene’s test, which was not significant either for 
the seven dependent variables (p. > .05). This confirmed the homogeneity assumption was 
not violated except for the sixth variable. In addition, population normality was also 
addressed because the three groups did not show skewness or kurtosis as the calculated 
values approached zero. Using a Kolmogorov-Smimov statistic with a Lilliefors significance 
level also resulted in values greater than 0.05, which assumed the three groups had been 
selected from a normally distributed population (Coakes & Steed, 2007). 
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Sum of Squares 

df 

Mean Square 

F 

Sig. 

OvCarSk 

Between Groups 

.158 

2 

.079 

.036 

.965 


Within Groups 

79.431 

36 

2.206 



CorsPlan 

Between Groups 

2.927 

2 

1.463 

.154 

.858 


Within Groups 

342.971 

36 

9.527 



ClassTea 

Between Groups 

.310 

2 

.155 

.045 

.956 


Within Groups 

122.665 

36 

3.407 



LearnAss 

Between Groups 

3.213 

2 

1.607 

.125 

.883 


Within Groups 

464.018 

36 

12.889 



ClassRes 

Between Groups 

1.190 

2 

.595 

.254 

.777 


Within Groups 

84.502 

36 

2.347 



CareePres 

Between Groups 

1.707 

2 

.853 

.650 

.528 


Within Groups 

47.268 

36 

1.313 




Table 5: ANOVA test of difference between three groups 


As summarized in Table 5, actual data analysis shows non-significant ANOVA F 
ratios (p. > 0.05), which indicates no differences between the three programs regarding 
faculty career development as a result of program evaluation (Coakes & Steed, 2007). When 
ANOVA does not yield significant F ratios, there is no need to use post hoc tests of multiple 
comparisons, such as Scheffe, Tukey or Newman-Keuls. This is simply because we did not 
have significant differences in the first pace to use post hoc tests in order to locate which 
group outperfonned the other (Gall et ah, 2006). Having this in mind, the researchers, 
however, used the Scheffe test to examine the means of each two groups to confirm the 
ANOVA results. Table 6 shows no differences between or within groups, which confirms the 
ANOVA findings. 


Dependent 

Variable 

(I) group 

(J) group 

(I-J) Mean 
Difference 

W) 

Dependent 

Variable 

(I) group 

(J) group 

(I-J) Mean 
Difference 



1 = 

2 

.04902 

.996 


1 = 

2 

-.08824 

.998 


1 = 

3 

-.11765 

.980 


1 = 

3 

-.68824 

.891 

OvCarSk 

2 = 

1 

-.04902 

.996 

LearnAss 

2 = 

1 

.08824 

.998 

2 = 

3 

-.16667 

.966 

2 = 

3 

-.60000 

.927 


3 = 

1 

.11765 

.980 


3 = 

1 

.68824 

.891 


3 = 

2 

.16667 

.966 


3 = 

2 

.60000 

.927 


1 = 

2 

-.17647 

.989 


1 = 

2 

.19608 

.944 


1 = 

3 

-.67647 

.860 


1 = 

3 

-.27059 

.907 

CorsPlan 

2 = 

1 

.17647 

.989 

ClassRes 

2 = 

1 

-.19608 

.944 

2 = 

3 

-.50000 

.931 

2 = 

3 

-.46667 

.778 


3 = 

1 

.67647 

.860 


3 = 

1 

.27059 

.907 


3 = 

2 

.50000 

.931 


3 = 

2 

.46667 

.778 


1 = 

2 

-.11765 

.986 


1 = 

2 

-.33824 

.738 


1 = 

3 

-.21765 

.957 


1 = 

3 

-.48824 

.570 

ClassTea 

2 = 

1 

.11765 

.986 

CareePres 

2 = 

1 

.33824 

.738 

2 = 

3 

-.10000 

.992 

2 = 

3 

-.15000 

.954 


3 = 

1 

.21765 

.957 


3 = 

1 

.48824 

.570 


3 = 

2 

.10000 

.992 


3 = 

2 

.15000 

.954 


Table 6: Scheffe test of multiple comparisons 


The results indicate that the program evaluation process equally influenced faculty 
career development in the three programs. The null hypothesis stating no inter-program 
differences at 0.05 in career development as a result of program evaluation was therefore 
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accepted. This means the alternative hypothesis stating inter-program differences at 0.05 in 
career development as a result of program evaluation was subsequently rejected. 

Qualitative Results 

Development of Teacher Curricular Content-knowledge 

The influence of the program evaluation process on career skills of faculty members 
seemed positive. “I have never thought this headache (program evaluation) would be that 
useful. It acted like an intensive but a practical training course. I think program evaluation 
can be a unique career development mechanism because we learn as we do our work.” The 
program evaluation process improved the teaching faculty skills in various ways. “Now I can 
better plan my courses.” Faculty members benefited more from the skills they developed 
through the course specification element. “When I attended discussions about preparing a 
course specification to address program evaluation concerns, I thought it was a waste of time. 
Later, when I was asked to prepare a course specification for each of my courses according to 
the new course specification template, I discovered that I missed the opportunity to learn 
from those discussions.” 

Development of Teacher Content-knowledge and Pedagogical Skills 

On creating the first course specification, “I had to address several issues. This 
included determining course objectives, course learning outcomes, course topics and 
materials and allocating teaching topics to the semester weeks and classes.” The process of 
creating a course specification required each faculty member “to spell out the infonnation 
and skills our students are expected to achieve by the end of the course. I even had to spell 
out the expected teaching strategies that I would use to enable my students to achieve target 
skills and information.” The process also demanded that “I should detennine the assessment 
tools that I would use to check that my students attained the predetermined learning 
outcomes.” Creating a course specification was like a curriculum planning and design process 
since every faculty member “had also to align course and program learning outcomes. 
Because I had to address course objectives, content and teaching and assessment strategies, I 
was a curriculum designer more than a just a teacher. To be honest, like my colleagues, 
although I was grappling throughout to address all of those elements, I developed in practical 
ways that I did not have during my initial training and other formal training courses.” 


Development of Teacher Reflection 

Another element that helped faculty members to improve their career skills during the 
evaluation process was reflection through preparation of course reports. “Having prepared the 
course specification, I thought all has been done. The discussions with the evaluation team 
made it clear that everything we planned to do in the course specification had to be reported 
on in a course report at the end of the semester.” This led faculty members to “be on the 
guard about how I respond to the course report questions about achieving what I planned in 
the course specification.” This involved reporting “on whether I covered all the topics I set 
out in the course specification. In case, some topics were not covered, I had to explain the 
compensating actions that I made to address that.” The course report also required reporting 
“on whether the teaching strategies I set out in the course specification were effective. Any 
difficulties should have been reported on as well as the actions made to overcome those 
difficulties. This had to be done with all of the skills and infonnation set out in the course 
specification.” 
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Development of Teacher Assessment Skills 

The impact of the course report process was notable. “I do not just teach but I also 
assess how effective my teaching is and how to arrange for additional teaching to address 
weaknesses. Any part of the course that was not covered required actions by me since the 
students also report on this. This made me develop not only my pedagogical and learning 
assessment skills but also the content knowledge of the subject on a daily basis.” The course 
report further required faculty members to report on “student results and include and 
comment on students’ assessments of the course as a whole and my own performance as a 
teacher. Based on the issues raised, I have also to submit plans for course improvement. 
These have made me see the full picture and work on improving my own skills.” 

The program evaluation process was the catalyst for faculty members to initiate their 
own development. “If we had not been forced to address several issues relating to our 
courses, we would not have exerted that effort. For example, alongside assigned course and 
reference books, I had to include my own materials that I used during the course. I had to 
justify why I chose the books and references I set for my students. This made me use specific 
criteria against which I evaluate the sources I chose. I also learned how to select and design 
supplementary materials.” During the evaluation process, “I had to think of a number of 
teaching strategies that would enable me to realize course objectives. This was not confined 
just to selecting a set of teaching strategies but also to think of the procedures of how they 
would be implemented in the classroom.” The evaluation process had a very positive impact 
on faculty skills of assessing student learning outcomes. “I am now more able to write 
different kinds of test items, check test reliability and validate test content. My skills 
particularly developed regarding analyzing and interpreting test results because I had to 
comment on student scores and grades in the course report.” 


Development of Teacher Research Skills and Pressure Coping Strategies 

The program evaluation process further improved faculty classroom research skills. “I 
have never thought that surveying and reporting on students’ opinions of the course and my 
teaching would be part of my work as a teacher. I now feel able to design questionnaires to 
collect students’ opinions about the course elements and my teaching. I am able now to 
analyse and interpret their responses because I have to include and comment on them in the 
course report and future course planning.” The program evaluation process also improved 
faculty cooperation and coping with work pressures. “To keep up with the new developments 
and workloads, I had to share experiences with other colleagues. We helped one another to 
achieve the tasks assigned to us. We also encouraged each other to address the new 
responsibilities. We now participate in many program committees to achieve tasks, this was 
good for us.” 

Development of Managerial Skills 

The program evaluation process also improved program directors’ skills in ways very 
similar to those of faculty members. “This program review turned everything upside down. I 
mean I used to make all the decisions and supervise everything myself. I understood it that 
way. This review process showed it otherwise where I had to devolve responsibilities to 
committees. This was great because I saw it work. I learned how to coordinate rather than do 
program work.” Program directors also learned from preparing the program specification and 
report. “As a program director, I had to determine in very clear terms what our program seeks 
to help program clients to achieve. I had to determine the main skills and infonnation our 
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students are expected to achieve.” This required “me in cooperation and consultation with all 
program stakeholders to define program mission, goals and objectives, program learning 
outcomes, domain knowledge and skills, suggested teaching and assessment strategies and 
program structure and organization.” This also required each program director to “describe 
the processes used in selecting texts and reference materials, faculty recruiting procedures 
and development opportunities, student administration and support and program evaluation 
and improvement processes.” 

Program directors also “had to prepare a program report that comments on the same 
elements in course reports but in relation to the whole program.” The impact of this was 
positive. “I acknowledge I learned a lot. The program elements have become clearer to me. I 
know better than before how to improve each element because I have specific standards 
according to which I assess the performance of each component.” For each program element, 
“for example, teaching and learning, student admission, resources, relationship with the 
community and student services, I have standards against which I measure program 
perfonnance. This enabled me to draw improvement plans since I collect data about what 
needs to be done. We now adopt an ongoing strategy of data collection. This was not that 
systematic before.” 


Discussion 

This study examined the influence of imposed program evaluation on faculty and 
program director career development in three language-education programs. The quantitative 
findings answered the first research question in positive (How does the program evaluation 
process impact on career development?). Overall, the findings indicated the three language 
programs were influenced by program evaluation in the same manner. The program 
evaluation process improved faculty career skills in terms of course planning, classroom 
teaching, learning assessment, classroom research, and coping with career pressure. The 
qualitative findings also indicated improvement in both program directors and faculty 
members’ professional skills. These findings agreed with previous research conclusions that 
program evaluation brings about a positive impact on developing program stakeholder career 
skills (e.g., Byrnes, 2008; Carsten-Wickham, 2008; Chase, 2006; Gorsuch, 2009). The 
findings also confirm Norris’s (2009) conclusion that positive program evaluation 
experiences provide an opportunity for program stakeholders to leam in the workplace. 

What is striking about these findings is that the three programs showed almost typical 
improvements. Why was that? Although future researchers may examine this, the possible 
explanation pointed to the vice-deanship of quality, which supervised the evaluation process 
across the three programs. Moreover, the three programs collected their data by the same 
scales standardized by the NCAAA. In addition, the three programs were against the same 
NCAAA standards. Finally, the three programs shared the same college context. 

Both the quantitative (inferential part) and qualitative findings, however, provided a 
neutral answer to the second research question (How does the program evaluation process 
impact on career development in the three programs?). These findings confirm rather than 
contradict the descriptive statistics results in that the evaluation process brought about a 
positive impact on faculty career development. Moreover, although these findings concurred 
with, for example, those of Byrnes (2008), Carsten-Wickham (2008), Chase (2006) and 
Gorsuch (2009) about the positive impact of program evaluation on career development, the 
reported previous research findings did not examine inter-program differences. Future 
researchers may therefore examine the relationship between career development and program 
evaluation experiences across different programs. 
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The qualitative findings, however, indicate that evaluation programs as a strategy for 
career development in the workplace require huge resources. Engaging all program 
stakeholders in an evaluation program for skills improvement purposes requires several 
educational workshops, short training sessions, frequent discussions and meeting and 
coordination. These require financial resources to achieve necessary activities. In this study’s 
case, our institution paid for overtime work and experts who provided training and 
consultations. Change takes time. In our case, improving faculty and program director skills 
occurred over a two-year evaluation program to put theory into action and create evaluation 
program culture. An evaluation program- such as this one- necessitated institutional support 
at the levels of department, program, college, and university administrations. No doubt such 
variables cannot be overcome in many contexts. 


Conclusions, Recommendations and Limitations 

The present study concluded that program evaluation improved faculty career skills, 
which included course planning, classroom teaching, learning assessment, classroom 
research, and coping with career pressure. In contrast, no inter-program differences were 
found regarding the influence of program evaluation process on faculty and program 
directors' career development across the three programs. The study recommended program 
evaluation as a reflection in action strategy not only for faculty and program director 
development but also for institutional, program, staff and student development. 

Future researchers may study the influence of program evaluation on administrative 
staff development, student learning, career satisfaction and institution improvement. 
Researchers may also examine the influence of program evaluation, especially imposed ones, 
on stakeholder attitudes toward the program evaluation process itself. Some caveats should, 
however, be considered before any generalization of the present findings into other contexts. 
The first caveat concerns data collection since it was gathered through self-reporting 
measures. Quantitative ability measures would yield not only important but also precise data 
about the exact developments in the skills of program directors and faculty members. 
Research designs need to use systematic observation at the beginning and at the end of the 
evaluation process. 
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Appendix: Program Evaluation Impact on Career Development 

This scale is used to collect your opinion of the program review process influence on the 
development of your career skills. You will find statements about each program element. 
Please read each one and circle the response (1, 2, 3, 4 or 5) that tells HOW TRUE OF YOU 
THE STATEMENT IS. 

1= Never or almost never true. 2= Usually not true. 3= Somewhat true. 

4= Usually true. 5= Always or almost always true. 


Section 1: Program evaluation impact on your overall career skills. 


1 . 

It improved my overall teaching skills. 

1 2 3 4 5 

2. 

It improved my domain knowledge. 

1 2 3 4 5 

3. 

It improved my skills of assessing classroom learning. 

1 2 3 4 5 


Section 2: Program evaluation impact on your course planning skills. 


4. 

I learned how to set precise course objectives. 

1 2 3 4 5 

5. 

I learned how to prepare the course specifications. 

1 2 3 4 5 

6. 

I learned how to prepare the course report. 

1 2 3 4 5 

7. 

I learned how to align course and program learning outcomes. 

1 2 3 4 5 

8. 

I learned how to fit course topics to teaching hours. 

1 2 3 4 5 

9. 

I learned how to design course materials. 

1 2 3 4 5 

10.1 learned how to evaluate and select course materials. 

1 2 3 4 5 


Section 3: Program evaluation impact on your classroom teaching skills. 

11.1 have become more able to spell out my teaching philosophy. 

1 2 3 4 5 

12.1 have become more able to set precise lesson objectives. 

1 2 3 4 5 

13.1 have become more able to align course and program learning objectives. 

1 2 3 4 5 

14.1 have become more able to align classroom teaching with exam content. 

1 2 3 4 5 

15.1 used various teaching strategies. 

1 2 3 4 5 


Section 4: Program evaluation impact on your skills of assessing learning outcomes. 


16. My skills of writing test items improved. 

1 2 3 4 5 

17. My skills of test design improved. 

1 2 3 4 5 

18.1 learned how to check test reliability. 

1 2 3 4 5 

19.1 learned how to validate test content. 

1 2 3 4 5 

20.1 learned how to analyze test results. 

1 2 3 4 5 

21.1 learned how to interpret test results. 

1 2 3 4 5 

22.1 learned how to create question banks. 

1 2 3 4 5 

23.1 used various assessment tools. 

1 2 3 4 5 

Section 5: Program evaluation impact on your skills of classroom research. 

24.1 learned how to survey student opinions. 

1 2 3 4 5 

25.1 learned how to analyze surveys. 

1 2 3 4 5 

26.1 learned how to interpret surveys. 

1 2 3 4 5 

27.1 learned how to incorporate student assessments into course planning. 

Section 6: Program evaluation impact on your career pressure. 

1 2 3 4 5 

28.1 learned how to work under pressure. 

1 2 3 4 5 

29.1 learned how to cope with team work. 

1 2 3 4 5 

30.1 learned how to address student concerns. 

1 2 3 4 5 
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